Image processing device, image processing method and program

ABSTRACT

According to an illustrative embodiment, an image processing device is provided. The image processing device includes a foreground selection processing circuit to select at least one foreground image that has been separated from a source image; a background selection circuit to select at least two display background images from at least one background image that has been separated from the source image; and a combination circuit to combine the at least one selected foreground image with the at least two display background images to generate a plurality of combined images, wherein at least one of the plurality of combined images does not appear in the source image.

BACKGROUND

The present disclosure relates to an image processing device, an imageprocessing method and a program, and in particular, to an imageprocessing device, an image processing method and a program that arecapable of realizing a novel reproduction effect.

A reproduction method in the related art is such that when reproducing astill image, one part of an imaging target object that is imaged intothe still image is reproduced as if it were in motion.

For example, in Japanese Unexamined Patent Application Publication No.2011-66717 is disclosed a reproduction method in which a region, onepart of a sequence of images according to the passage of time, isstopped as a static display region at a predetermined time, and otherregions are reproduced as dynamic display regions. However, in thetechnology disclosed in Japanese Unexamined Patent ApplicationPublication No. 2011-66717, any one of an image in a region that issuperimposed and an image in a region onto which the region issuperimposed is stationary in terms of a position in which drawing isprovided with respect to a display surface.

Furthermore, in Japanese Unexamined Patent Application Publication No.2010-124115 is disclosed a reproduction method in which a moving imagethat is configured from consecutive images is used as a raw material, orrelevant consecutive still images and a still image unrelated to therelevant consecutive still images are used as the raw materials, andthus one part of the raw material is reproduced as if it were in motion.However, in the technology disclosed in Japanese Unexamined PatentApplication Publication No. 2010-124115, only the image that issuperimposed is dynamic in terms of a position in which rendering isprovided with respect to a display surface. Furthermore, an unrelatedimage is used in an image onto which an image is superimposed and in theimage that is superimposed.

SUMMARY

Incidentally, the reproduction method is demanded by which a novelreproduction effect that is not possible with the technologies disclosedin Japanese Unexamined Patent Application Publication Nos. 2011-66717and 2010-124115 is realized.

It is desirable to realize a novel reproduction effect.

In view of the above, the embodiments of the present technology areprovided. According to an illustrative embodiment, an image processingdevice includes a foreground selection processing circuit to select atleast one foreground image that has been separated from a source image;a background selection circuit to select at least two display backgroundimages from at least one background image that has been separated fromthe source image; and a combination circuit to combine the at least oneselected foreground image with the at least two display backgroundimages to generate a plurality of combined images, wherein at least oneof the plurality of combined images does not appear in the source image.

Accordingly, a novel reproduction effect can be realized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view for describing first image processing.

FIG. 2 is a block diagram illustrating a configuration example of animage processing device according to one embodiment, to which thepresent technology is applied.

FIG. 3 is a flowchart describing the first image processing.

FIG. 4 is a view for describing second image processing.

FIG. 5 is an image, sufficiently large in view angle, which is made frommultiple sheets of image that makes up a moving image.

FIG. 6 is a flowchart describing second image processing.

FIG. 7 is a flowchart describing a modification example of the secondimage processing.

FIG. 8 is a view for describing third image processing.

FIG. 9 is a flowchart describing the third image processing.

FIG. 10 is a block diagram illustrating a configuration example of adigital video camera.

FIG. 11 is a block diagram illustrating a configuration example of acomputer to which the present technology is applied, according to oneembodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Specific embodiments to which the present technology is applied aredescribed in detail below referring to the drawings,

First, image processing to which the present technology is applied isdescribed referring to FIG. 1.

A source material image 11 (or “raw-material image 11”), a raw materialon which the image processing is performed is illustrated on theuppermost portion in FIG. 1, and a horizontally-long panoramic imagehaving an aspect ratio is used as the raw-material image 11 in anexample in FIG. 1. Then, the foreground and the background are definedwith respect to an imaging target object that is imaged into theraw-material image 11, and the raw-material image 11 is separated into aforeground image and a background image, any of which is a target forscroll reproduction.

For example, as illustrated in the second portion of FIG. 1 from above,a balloon that is imaged into the raw-material image 11 is defined as aforeground image 12 and a scene that is imaged into the raw-materialimage 11 is defined as a background image 13, thereby separating theraw-material image 11 into the foreground image 12 and the backgroundimage 13. Then, in the example in FIG. 1, only the background image 13is the target for scroll reproduction.

Next, as illustrated on the third portion of FIG. 1 from above, adisplay region 14, which specifies a region that is defined as a displaytarget when performing the scroll reproduction, is set with respect tothe background image 13 that is the target for scroll reproduction. Forexample, when the scroll reproduction is assumed to be performed fromthe left side to the right side of the background image 13, the displayregion 14, as illustrated, is set at the left end of the backgroundimage 13. Then, a region that is prescribed by the display region 14 isseparated (or “extracted” or “cropped”) as a display background image 15from the background image 13, and a combination image 16 is generated bycombining the display background image 15 and the foreground image 12that is not defined as the target for scroll reproduction.

Then, according to the number of frames at the time of the scrollreproduction, the display region 14 is set with respect to thebackground image 13 in such a manner that a position of the displayregion 14 is gradually moved to the right and the foreground image 12 iscombined with respect to each frame.

As illustrated on the fourth portion of FIG. 1 from above, thecombination image 16 is generated in which the foreground image 12 isarranged all the time within the display region 14 while scrolling thebackground image 13. For example, in a combination image 16 T(0) at apoint in time T0 when reproduction of the combination image 16 isstarted, the left end of the background image 13 is extracted. Then, ina combination image 16T(i) at a point in time Ti, the background image13 at a position corresponding to the point in time Ti is extracted, andin a combination image 16T(n) at a point in time Tn when thereproduction of the combination image 16 is ended, the right end of thebackground image 13 is extracted. Furthermore, in the combination image16, when combining the foreground image 12, coordinates (x, y) at whichthe foreground image 12 is arranged are stationary between thecombination images 16T(0) to 16T(n).

By generating the total number n of frames, the combination images 16T(0) to 16T(n) in this manner, a moving image is generated that has aneffect in which the reproduction occurs as if the balloon, theforeground image 12, were displayed all the time, and the scene, thebackground image 13 were displayed while scrolled according to amovement of the display region 14. In other words, an effect can beobtained in which the reproduction occurs if the static imaging targetobject were imaged into the dynamic background.

Next, FIG. 2 is a block diagram illustrating a configuration example ofan image processing device according to one embodiment, to which thepresent technology is applied.

An image processing device 21 performs the image processing on theraw-material image 11 that is input as the raw material, generates thecombination image 16, and retains the result in a memory 22. The imageprocessing device 21, as illustrated in FIG. 2, is configured to includea count processing circuit 31 (or “course processing unit” 31), aseparation circuit 32 (or “separation unit” 32), a determinationprocessing circuit 33 (or “determination processing unit” 33), aforeground selection processing circuit 34 (or “foreground selectionprocessing unit” 34), a background extraction circuit 35 (or “backgroundextraction unit” 35) and a combination circuit 36 (or “combination unit”36).

The count processing circuit 31 counts a count value T(i) to count thenumber of frames of the combination image 16 that is generated in theimage processing device 21, and performs count processing to make acomparison with the total number n of frames of the combination image 16that are generated in the image processing device 21.

For example, an image that is recorded in a record circuit (or “recordunit”) not illustrated is input as the raw-material image 11 into theseparation circuit 32. Then, the separation circuit 32 separates theraw-material image 11 into the foreground image 12 and the backgroundsight image 13, supplies the foreground image 12 to the foregroundselection processing section 34 and supplies the background image 13 tothe background extraction circuit 35.

For example, the separation circuit 32 detects an edge of a photographtarget object that is imaged into the raw-material image 11, defines aregion into which the photograph target object is imaged, as theforeground image 12 and defines the other regions as the backgroundimage 13. Furthermore, for example, a user may appoint a region of theraw-material image 11 that is defined as the foreground image 12, byoperating an operation circuit (or “operation unit”) not illustrated.The separating circuit 32 defines the region appointed by the user asthe foreground image 12 and defines the other regions as the backgroundimage 13. Furthermore, the separation circuit 32 can supplement theregion into which the foregoing image 12 is imaged in the backgroundimage 13, with an image adjacent to that region.

The determination processing circuit 33 performs determinationprocessing, such as determining a size and a position of the displayregion 14 that is set with respect to the background image 13, ordetermining the total number n of frames of the combination image 16that the count processing circuit 31 refers to in the counterprocessing. For example, when performing the scroll reproduction that isdescribed referring to FIG. 1, the determination processing circuit 33determines the size of the display region 14 in accordance with a heightof the raw-material image 11. Then, the determination processing circuit33 determines the left end of the raw-material image 11 as the firstposition of the display region 14 and determines the right end of theraw-material image 11 as the last position of the display region 14.

The foreground selection processing circuit 34 selects (determines) theforeground image 12 that is used as the foreground in the combinationimage 16. For example, when using the panoramic image as theraw-material image 11 as illustrated referring FIG. 1, because theforeground image 12 is the only one, the foreground selection processingcircuit 34 selects the foreground image 12. Furthermore, for example, asdescribed below referring to FIG. 4, when using the moving image as theraw-material image 11, the foreground selection processing circuit 34selects the foreground image 12 that is combined into the combinationimage 16, from the multiple foreground images 12 that are imaged intothe raw-material image 11.

The background extraction circuit 35 sets the display region 14 withrespect to the background image 13 supplied from the separation circuit32, according to the determination by the determination processingcircuit 33, extracts the display background image 15 from the backgroundimage 13 based on the display region 14, and supplies the result to thecombination circuit 36.

At this time, the background extraction circuit 35 sets the displayregion 14 to a position that is according to the count value T obtainedby the count processing circuit 31. In other words, the backgroundextraction circuit 35 sets the display region 14 to the first positionthat is determined by the determination processing circuit 33, accordingto the count value T(0), and sets the display region 14 to the castposition that is determined by the determination processing circuit 33,according to the count value T(n). Then, for example, the backgroundextraction circuit 35 sets the display region 14 that is according tothe count value T(i), to an i-th position that results from, equallydividing spacing between the first position and the last position or thedisplay region 14 by the total number n of frames.

In other words, the background extraction circuit 35 extracts thedisplay background image 15T(i) from the background image 13, based onthe display region 14 that is set according to the count value T(i) andsupplies the display background image 15T(i) to the combination circuit36.

The combination circuit 36 combines the foreground image 12 that issupplied from the foreground selection processing circuit 34, in such amanner that the foreground image 12 is superimposed onto the backgroundimage 13T(i) that is supplied from the background extraction circuit 35,thereby generating the combination image 16T(i). Then, the combinationcircuit 36 supplies the generated combination image 16T(i) to the memory22 and the generated combination image 16T(i) is retained in the memory22.

The image processing device 21 is configured in this manner. Thebackground extraction circuit 35 supplies the display background images15T(0) to 15T(n) to the combination circuit 36, according to the countvalues T(0) to T(n). The combination circuit 36 generates (combines) themoving image that is made from the combination images 16T(0) to 16T(n).Therefore, by sequentially reading the combination images 16T(0) to16T(n) out of the memory 22 and reproducing the read-out combinationimages, an effect can be obtained in which the reproduction occurs as ifonly the scene of the background image 13 were moved with the balloon,the background image 12, remaining stationary.

Next, the image processing by the image processing device 21 isdescribed referring to a flowchart of FIG. 3.

For example, when the raw-material image 11 is supplied to the imageprocessing device 21, the processing is started. In Step S11, theseparation circuit 32 separates the raw-material image 11 into theforeground image 12 and background image 13, supplies the foregroundimage 12 to the foreground selection processing circuit 34, and suppliesthe background image 13 to the background extraction circuit 35.

In Step S12, the count processing section 31 initializes the count valueT(i) to count the number of frames of the combination image 16 and setsthe count value T(i) to 0.

In Step S13, the determination processing circuit 33 determines the sizeand the position of the display region 14 that is set with respect tothe background image 13. For example, if a still image is defined as theraw material, the determination processing circuit 33 determines thesixe of the display region 14 in such a manner that the size of thedisplay region 14 is smaller than the size of the background image 13.

In Step S14, the count processing circuit 31 determines whether or notthe current count value T(i) is less than the total number n of frames,and if it is determined that the current count value I(i) is less thanthe total number n of frames (i<n), the processing proceeds to Step S15.In other words, the count processing section 31 compares the totalnumber n of frames with the current count value T(i), and as long as thecombination image 16T(i) generated at the current count value T(i) doesnot reach the combination image 16T(i)n that is generated correspondingto the total number n of frames, the processing continues.

In Step S15, the foreground selection processing circuit 34 determinesthe foreground, image 12 that is used as the foreground in thecombination image 16.

In Step S16, the background extraction circuit 35 sets the displayregion 14 of which the size is determined by the determinationprocessing circuit 33 in Step S13, to a position corresponding to thecurrent count value T(i), with respect to the background image 13. Then,the background extraction circuit 35 extracts the display backgroundimage 15T(i) from the background image 13 according to the displayregion 14 and supplies the result to the combination circuit 36.

n Step S17, the combination circuit 36 combines the foreground image 12selected by the foreground selection processing circuit 34 in Step S15with the display background image 15T(i) extracted by the backgroundextraction circuit 35 from the background image 13 in Step S16, therebygenerating the combination image 16T(i).

In Step S18, the combination circuit 36 supplies the combination image16T(i) generated in Step S17 to the memory 22 and the combination image16T(i) is retained in the memory 22.

In Step S19, the count processing section 31 increments the count valueT(i) only by one, and the processing returns to Step S14. Subsequently,the same processing is repeatedly performed.

Thereafter, in Step S14, if the count processing circuit 31 determinesthat the current count value T(i) is not less than the total number n offrames, that is, that the current count value T(i) is the total number nof frames or more (i≧n), the processing is ended,

As described above, when the still image (for example, the panoramicimage) is supplied as the raw-material image 11, the image processingdevice 21 can generate the combination image 16 that is reproducible insuch a manner that the imaging target object is imaged all the timewhere the scene of the still image moves. In other words, bysequentially reading the combination images 16T(0) to 16T(n) out of thememory 22 and reproducing the read out combination images, a noveleffect can be obtained in which the reproduction occurs as if only thescene of the background image 13 were moved with the balloon, thebackground image 12, remaining stationary.

In other words, by using the foreground image 12 and the backgroundimage 13 that are separated from the raw-material image 11, arelationship is present between the superimposed foreground image 12 andthe background image 13 onto which the foreground image 12 issuperimposed, and furthermore the position on which the background image13 is displayed moves in the combination image 16. Thus, the imageprocessing device 21 can accomplish the new reproduction effect that isdifferent from the reproduction effect in the related art.

Moreover, in addition to performing the image processing that uses thestill image as the raw-material image 11, which is described referringto FIG. 1, the image processing device 21 can perform the imageprocessing that uses the moving image as the raw-material image 11. Theimaging process that uses the still image as the raw-material image 11,which is described above is hereinafter referred to as first imageprocessing for the sake of convenience.

Next, second image processing that uses the moving image as theraw-material image 11 in the image processing device 21 is describedreferring to FIG. 4.

On the uppermost portion in FIG. 4, raw-materials 11T(0) to 11T(n) areillustrated that make up the moving image, raw materials on which theimage processing is performed. Then, the foreground and the backgroundare defined with respect to the raw-material images 11T(0) to 11T(h),and the raw-material images 11T(0) to 11T(h) are separated into theforeground images and the background images.

For example, as illustrated on the second portion of FIG. 4 from above,a balloon that is imaged into the raw-material images 11T(0) and 11T(b)are defined as foreground images 12T(0) to 12T(b). Furthermore, a scenethat is imaged into the raw-material images 11T(0) to 11T(h) are definedas background images 13T(0) to 13T(h). Then, the raw-material images11T(0) to 11T(b) into which the balloon is imaged are separated into theforeground images 12T(0) to 12T(b) and the background images 13T(0) to13T(b). Moreover, raw-material images 11T(c) to 11T(h) into which theballoon is not imaged, as they are, are used as background images 13T(c)to 13T(h),

Furthermore, the display region 14 is determined as having the same sizeas the background images 13T(0) and 13T(h), and all regions of each ofthe background images 13T(0) to 13T(h) are displayed.

Then, the foreground images 12T(0) to 12T(b) are combined with thebackground images 13T(0) to 13T(h) in such a manner that the foregroundimages 12T(0) to 12T(b) are superimposed onto the background images13T(0) to 13T(h). Here, the last foreground image 12T(b) of theforeground images 12T(0) to 12T(b) is combined with respect to thebackground images 13T(c) to 13T(h). In other words, the foreground image12T(b) is used as being of a still image. Moreover, a position of theforeground image 12T(b) may is properly moved that is superimposed ontothe background images 13T(c) to 13T(h).

As illustrated On the third portion of FIG. 4 from above, this makes itpossible to generate combination images 16T(0) to 16T(h) in which theforeground image 12 is arranged all the time with respect to thebackground images 13T(c) to 13T(h).

Furthermore, as a modification example of the second image processing,the foreground image into which the imaging target object is imaged mostexcellently among the foreground images 12T(0) to 12T(b) may be combinedwith respect to the background images 13T(c) to 13T(h). For example, noblurring, no shaking, big smile, no closed eyes, brightness and the likeare used as criteria for determining the best imaging of the imagingtarget object into the foreground image. Furthermore, the best imagingof the imaging target object may be automatically selected or may beselected by a user through the use of information on the foregroundimage 12 itself and information applied to the foreground image 12.Regarding selection of the foreground image based on brightness, itshould be noted that the foreground image may be selected according to alevel of brightness that provides the best foreground image, so as toavoid selection of an image with a brightness that is too high or toolow.

Moreover, since the background images 13T(c) to 131(h) are notstationary, that is, since the background moves in the raw-materialimages 11T(0) to 11T(b), the effect in which the reproduction occurs asif the static imaging target object were imaged into the dynamicbackground can be obtained in the second image processing.

At this point, in FIG. 5, an image is illustrated in which theraw-material images 11T(0) to 11T(h) are arranged two-dimensionally andare displayed.

The raw-material images 11T(0) to 11T(h) are matched to the image, andthe raw-material images 11T(0) to 11T(h) are combined with regions ofthe image with which the raw-material images 11T(0) to 11T(h) agree,respectively, in an overlapping manner. Moreover, a technology, asillustrated in FIG. 5, which generates the image 17, sufficiently largein view angle, from the multiple sheets or image that make up the movingimage is disclosed in detail, for example, in Japanese Unexamined PatentApplication Publication No. 2009-077363 and US Patent ApplicationPublication No. 2010/0066860, both filed by the applicant of the presentapplication, and both hereby incorporated by reference herein.

The background images 13T(0) to 13T(h) are the moving images of which animage capture position is not stationary, but the image that issufficiently as large in view angle as the image 17 can be generated byusing the background images 13T(0) to 13T(h). Moreover, the image 17 isnot only generated from the multiple sheets of image that make up themoving image, but the image 17 may also be created by using the stillimages, obtained by consecutive shooting, as the raw materials.

Next, the second image processing by the image processing device 21 isdescribed referring to a flowchart of FIG. 6.

For example, when the raw-material images 11T(0) to 11T(h) that make upthe moving image are provided to the image processing device 21, theprocessing is started. In Step S21, the separation circuit 32 separatesthe raw-material images 11T(0) to 11T(b), into which the balloon isimaged, into the foreground images 12T(0) to 12T(b) and the backgroundimages 13(0) to 13T(b). Then, the separation circuit 32 supplies theforeground image 12T(0) to 12T(b) to the foreground selection processingcircuit 34 and supplies the background image 13T(0) to 13T(b) to thebackground extraction circuit 35. Furthermore, the separation circuit 32supplies the raw-material images 11T(c) to 11T(n), as they are, intowhich the balloon is not imaged, as the background image 13T(c) to13T(h) to the background extraction circuit 35.

In Step S22, the count processing section 31 initializes the count valueT(i) to count the number of the frames of the combination image 16 andsets the count value T(i) to 0.

In Step S23, the determination processing circuit 33 determines the sizeand the position of the display region 14 that is set with respect tothe background images 131(0) to 13T(h). For example, if the moving imageis defined as the raw material, the determination processing circuit 33determines the size and the position of the display region 14 in such amanner that the display region 14 agrees with each of the backgroundimages 13T(0) to 13T(h) in terms of size and position, that is, in sucha manner that all regions of each of the background images 13T(0) to13T(h) are displayed.

It should be noted that Step 23 can be bypassed in the event that thesource image is a moving image. For example, when the source image is amoving image, one or more frames of the source image can be used asbackground images directly, without any need for extraction. Therefore,in such a case, there is no need to set a display region for use inextraction.

In Step S24, the count processing circuit 31 determines whether or notthe current count value T(i) is less than the total number n of frames,and if it is determined that the current count value T(i) is less thanthe total number h of frames (i<h), the processing proceeds to Step S25.

In step S25, the foreground selection processing circuit 34 determineswhether or not the imaging target object that is defined as theforeground image 12T(i) is imaged into the raw-material image 11T(i), aprocessing target object that is according to the current count valueT(i). For example, in Step S21, if the row-material image 11T(i) isseparated into the foreground image 12T(i) and the background image13T(i), the foreground selection processing circuit 34 determines thatthe imaging target object defined as the foreground image 12T(i) isimaged into the raw-material image 11T(i), the processing target object.

In Step S2S, the processing proceeds to Step 26 if the foregroundselection processing circuit 34 determines that the imaging targetobject defined as the foreground image 12T(i) is imaged into theraw-material image 11T(i), the processing target object. In Step S26,the foreground selection processing circuit 34 selects the imagingtarget object that is imaged into the raw-material image 11T(i), theprocessing target object, that is, the foreground image 12T(i) separatedfrom the raw-material image 11T(i), and determines the selectedforeground image 12T(i) as being combined into the combination image16T(i).

In Step 27, the background extraction circuit 35 extracts the displaybackground image 15T(i) from the background image 13T(i) according tothe display region 14 and supplies the result to the combination circuit36. Moreover, in the second image processing, the display region 14 isdetermined in Step 323 in such a manner as to agree with the backgroundimage 13T(i), Because of this, the background extraction circuit 35supplies all regions of the background image 13T(i) as the displaybackground image 15T(i) to the combination circuit 36 and supplies thebackground image 13T(i), as it is, as the background 15T(i) to thecombination circuit 36.

In Step S28, the combination circuit 36 combines the foreground image12T(i) selected by the foreground selection processing circuit 34 inStep S26 with the display background image 15T(i) supplied from thebackground extraction circuit 35 in Step S27, thereby generating thecombination image 16T(i).

In Step S29, the combination circuit 36 supplies the combination image16T(i) generated in Step S28 to the memory 22, for retention in there.

In Step S30, the count processing section 31 increments the count valueT(i) only by one, and the processing returns to Step S24. From Step S24onwards, the same processing is repeatedly performed.

On the other hand, in Step S25, the processing proceeds to Step 31 ifthe foreground selection processing circuit 34 determines that theimaging target object defined as the foreground image 12T(i) is notimaged into the raw-material image 11T(i), the processing target object.

In Step S31, the foreground selection processing circuit 34 determineswhether or not the foreground image 12 is separated in the raw-materialimages 11T(0) to 11T(i−1) before the raw-material image 11T(i), theprocessing target object.

In Step S31, if the foreground selection processing circuit 34determines that the foreground image 12 is separated in the raw-materialimages 11T(0) to 11T(i−1) before the raw-material image 11T(i), theprocessing target object, the processing proceeds to S32. In Step S32,the foreground selection processing circuit 34 determines that theforeground image 12 that is separated from the immediately precedingraw-material image 11, among the foreground images 12 that are separatedfrom the raw-material images 11T(0) to 11T(i−1), is combined into thecombination image 16T(i).

After performing the processing in Step S32, the processing proceeds toStep S27. From Step S27 onwards, the processing described above isperformed.

On the other hand, in Step S31, if the foreground selection processingcircuit 34 determines that the foreground image 12 is not separated inthe raw-material images 11T(0) to 11T(i−1) before the raw-material image11T(i), the processing target object, the processing proceeds to Step327 without selecting the foreground image 12 that is combined into thecombination image 16T(i). From Step S27 onwards, the processingdescribed above is performed. That is, in this case, because the imagingtarget object defined as the foreground image 12 is not imaged,exception processing is performed that does not combine the foregroundimage 12.

Thereafter, in Step S24, if the count processing circuit 31 determinesthat the current count value T(i) is not less than the total number h offrames, that is, that the current count value T(i) is the total number hof frames or more (i≧h), the processing is ended.

As described above, when the moving image is supplied as theraw-material image 11, the image processing device 21 can generate thecombination image 16 that is reproducible in such a manner chat theimaging target object is imaged all the time against one background ofthe moving image. In other words, by sequentially reading thecombination images 16T(0) to 16T(h) out of the memory 22 and reproducingone read-out combination images, the novel effect can be obtained inwhich the reproduction occurs as if the balloon, the foreground image 12were displayed all the time, and only the scene, the background image 13were moved.

Moreover, the image processing device 21, as described above, maycombine the foreground image into which the imaging target object isimaged most excellently among the foreground images 12T(0) to 12T(b),with respect to the background images 13T(c) to 13T(h).

That is, a modification example of the second image processing by theimage processing device 21 is described referring to a flowchart of FIG.7. Moreover, the flowchart of FIG. 7 illustrates processing performedfrom Step S24 to before Step S27 in FIG. 6.

That is, if the count processing section 31 determines in Step S24 inFIG. 6 that the current count value T(i) is less than the total number hof frames (i<h), the processing proceeds to Step S41.

In step S41, the foreground selection processing circuit 34 determineswhether or not the imaging target object that is defined as theforeground image 12T(i) is imaged into the raw-material image 11T(i), aprocessing target object that is according to the current count valueT(i), in the same manner as in Step S25 in FIG. 6.

In Step S41, if the foreground selection processing circuit 34determines that the imaging target object defined as the foregroundimage 12T(i) is imaged into the raw-material image 11T(i), theprocessing target object, the processing proceeds to Step 42. In stepS42, the foreground selection processing circuit 34 determines whetheror not the foreground image 12 is present, that is a candidate that iscombined into the combination image 16T(i).

In Step S42, the processing proceeds to Step S43 if the foregroundselection processing circuit 34 determines that the foreground image 12is present that is the candidate that is combined into the combinationimage 16T(i). In Step S43, the foreground selection processing circuit34 determines whether or not the imaging target object, the foregroundimage 12T(i) that is imaged into the raw-material image 11T(i), theprocessing target object, is more excellent, that is, is better inimaging, than the imaging target object, the foreground image 12 that isthe candidate.

In Step S43, the processing proceeds to Step S44 if it is determinedthat the imaging target object, the foreground image 12T(i) that isimaged into the raw-material image 11T(i), the processing target object,is not more excellent than the imaging target object, the foregroundimage 12 that is the candidate. In Step S44, the foreground selectionprocessing circuit 34 selects the foreground image 12, which is thecandidate, as being combined into the combination image 16T(i).

On the other hand, the processing proceeds to Step S45 if it isdetermined in Step S43 that the imaging target object, the foregroundimage 12T(i) that is imaged into the raw-material image 11T(i), theprocessing target object, is more excellent than the imaging targetobject, the foreground image 12 that is the candidate, or if it isdetermined in Step S42 that the foreground image 12, the candidate thatis combined into the combination image 16T(i), is not present. In StepS45, the foreground selection processing circuit 34 selects theforeground image 12T(i) that is imaged into the raw-material image11T(i), the processing target object, as being combined into thecombination image 16T(i).

Then, after performing the processing in Steps S44 or S45, or in StepS41, if it is determined that the imaging target object defined as theforeground image 12T(i) is not imaged into the raw-material image11T(i), the processing target object, the processing proceeds to Step 27in FIG. 6.

As described above, the image processing device 21 compares the imagingtarget object, the foreground image 12T(i), which is imaged into theraw-material image 11T(i), the processing target object, and the imagingtarget object, the foreground image 12 that is the candidate. Thus, theimage processing device 21 can generate the combination image 16 byusing the foreground image 12 that turns out to be a more excellentimaging target object.

Next, third image processing that is performed in the image processingdevice 21 is described referring to a flowchart of FIG. 8.

On the uppermost portion in FIG. 8, a leading frame that makes up themoving image is illustrated as the raw-material image 11. Furthermore,as illustrated by a region indicated by hatching in the second portionof FIG. 8 from above, a person and a bicycle that are imaged into theraw-material image 11 are defined as the foreground image 12 and areseparated from the raw-material image 11. Furthermore, as illustrated bya region indicated by the hatching in the third portion of FIG. 8 fromabove, a scene that is imaged into the raw-material image 11 is definedas the background image 13 and is separated from the raw-material image11. Furthermore, in the same manner as in the second image processing,the display region 14 is determined as having the same size as thebackground image 13, and all regions of the background image 13 aredisplayed.

Then, in the third image processing, the combination image is generatedin such a manner that both of the foreground image 12 and the backgroundimage 13 move and the foreground image 12 and the background image 13are displayed at fixed coordinates. At this time, for example, an effectin which the reproduction occurs as if flow of the background image 13over time were different from flow of the foreground image 12 over timecan be obtained by changing a reproduction speed of the background image13 relative to a reproduction speed of the foreground image 12. Forexample, when she foreground image 12 is reproduced at the samereproduction speed as the raw-material image 11, and the backgroundimage 13 is reproduced at double the reproduction speed at which theraw-material image 11 is reproduced, the combination image in which thebackground image 13 moves is generated as if the foreground image 12moved at the double speed. That is, when a multiplication factor of thereproduction speed of the background image is doubled, the foregroundimage appears to move at double speed. One way to control thereproduction speed of the foreground image is by controlling the rate ofreproduction of the image frames making up the foreground image.Similarly, one way to control the reproduction speed of the backgroundimage is by controlling the rate of reproduction of the image framesmaking up the background image. Thus, for example, to reproduce theforeground image at a different speed from the background image, thereproduction frame rate of the foreground image may be set differentfrom the reproduction frame rate of the background image.

Next, the third image processing by the image processing device 21 isdescribed referring to a flow chart of FIG. 9.

For example, processing is started that supplies the first frame, whichmakes up the moving image, as the raw-material image 11 to the imageprocessing device 21. In Step S51, the separation circuit 32 starts theprocessing that separates the raw-material image 11 into the foregroundimage 12 and the background sight image 13, supplies the foregroundimage 12 to the foreground selection processing section 34 and suppliesthe background image 13 to the background extraction circuit 35.Furthermore, frames making up the moving image are sequentially suppliedto the separation circuit 32, and the separation circuit 32 sequentiallyseparates the frames as the raw-material image 11.

In Step S32, the count processing section 31 initializes the count valueT(i) to count the number of the frames of the combination image 16 andsets the count value T(i) to 0.

The determination processing circuit 33 determines the total number N offrames of the moving image that is output, in Step S53 and determines areproduction multiplication factor a of the display background image 15(all regions of the background image 13 if the display region 14 isdetermined as having the same size as the background image 13) in StepS54. Moreover, the total number N of frames and the reproductionmultiplication factor a, for example, may be input based on thereproduction effect that a user wants and may be determined according tothe input of the reproduction effect.

n Step S55, the count processing circuit 31 determines whether or notthe current count value T(i) is less than the total number N of frames,and if it is determined that the current count value T(i) is less thanthe total number N of frames (i<N), the processing proceeds to Step S56.

In Step S56, the foreground selection processing circuit 34 selects theforeground image 127(i) that is separated from the raw-material 11T(i)that is according to the current count value T(i), and determines theselected foreground image 12T(i) as being combined into the combinationimage 16T(i).

In Step S57, the foreground selection processing circuit 35 selects thebackground image 13T(i×a) that is separated from the raw-material11T(i×a) that is according to a value that results from multiplying thecurrent count value T(i) by the reproduction multiplication factor a,and determines the selected background image 13T(i×a) as being combinedinto the combination image 16T(i).

In Step S58, the combination circuit 36 combines the foreground image127(i) determined by the foreground selection processing circuit 34 inStep S56 with the display background image 13T(i×a) determined by thebackground extraction circuit 35 in Step S57, thereby generating thecombination image 16T(i). At this time, the combination circuit 36 fixesa position, in which the foreground image 12T(i) is superimposed on thebackground image 13T(i×a), to the same position each time.

In Step S58, the combination circuit 36 supplies the combination image16T(i) generated in Step S58 to the memory 22, for retention in there,and generates the moving image as a result of the combination.

In Step S60, the count processing section 31 increments the count valueT(i) only by one, and the processing returns to Step S55. From Step S55onwards, the same processing is repeatedly performed.

Thereafter, the processing is ended in Step S55 if the count processingcircuit 31 determines that the current count, value T(i) is not lessthan the total number N of frames, that is, that the current count valueT(i) is the total number N of frames or more (i≧N).

As described above, when the moving image is supplied as the rawmaterial image 11, the image processing device 21 can generate thecombination image 16 that is reproducible in such a manner that theimaging target object that is reproduced at the same reproduction speedis imaged all the time against the background that is reproduced at thereproduction multiplication factor a. In other words, an effect in whichthe reproduction occurs as if only the background were fast forwardedcan be obtained by sequentially reading the frames making up thecombination image 16 out of the memory 22 and reproducing the read-outframes.

The reproduction multiplication factor a of the background image 13 hereis a multiplication factor for the reproduction speed of the movingimage that is used as the raw-material, and the multiplication factor isnot limited to integer multiplication. Furthermore, if the reproductionmultiplication factor a of the background image 13 is determined as avalue of less than 1, the background image 13 is reproduced at the speedequal to or less than that of the foreground image 12, and areproduction effect, like so-called slow reproduction, can be obtained.For example, interpolation can be performed on the foreground image orsource image, and the interpolated images can be combined with theimages which correspond to the reduced speed background image.Furthermore, the background image 13 may be reproduced at the samereproduction speed, and the foreground image 12 may be reproduced at thereproduction speed corresponding to the reproduction multiplication facta. In other words, the reproduction speed is set with respect to each ofthe foreground image 12 and the background image 13, and the combinationimage is generated in such a manner that the reproduction is performedat each of the reproduction speeds.

Moreover, each of the first to third image processing is one of theexamples. The combination image 16 may be generated in such a mannerthat only the foreground moves with the background remaining stationary.The combination image 16 may be generated in such a manner that only thebackground moves with the foreground remaining stationary. Furthermore,the combination image 16 may be generated in such a manner that theforeground and the background move individually.

Next, FIG. 10 is a block diagram illustrating a configuration example ofa digital video camera equipped with the image processing device 21.

As illustrated in FIG. 10, in addition to the image processing device 21and the memory 22 in FIG. 2, a digital video camera 41 is configured toinclude an imaging circuit 42 (or “imaging unit” 42), a basebandprocessing circuit 43 (or “baseband processing unit” 43), a switch 44, asignal output interface (I/F) 45, a display system I/F 46, an encoder47, a record processing circuit 48 (or “record, processing unit” 48), aswitch 49, a record circuit 50 (or “record unit” 50), a readoutprocessing circuit 51 (or “readout processing unit” 51), a decoder 52,an operation system I/F 53 and a system control circuit 54 (or “systemcontrol unit” 54).

Baseband processing is performed on an image captured by the imagingcircuit 42 in the baseband processing circuit 43, and the resultingimage is supplied to the image processing circuit 21 through the switch44 and is output through the signal output I/F 45 and the display systemI/F 46. If the image processing as described above is performed, theimage is supplied to the image processing device 21, and the imageprocessing device 21 performs the image processing on the image as theraw-material image 11.

The image processing device 21 outputs the combination image 16,obtained as a result of the image processing, to an external apparatusthrough the signal output I/F 45 or display a display apparatus (notillustrated) through the display system I/F 46. Furthermore, the imageprocessing device 21 supplies the combination image 16 to the encoder 47to encode the combination image 16, and the combination image 16 encodedby the encoder 47 is recorded by the record processing circuit 48 in therecord circuit 50 through the switch 49.

Furthermore, the readout processing circuit 51 read out the imagerecorded in the record circuit 50 through the switch 49, the read-outimage is supplied to the decoder 52 to decode the read-out image, andthe decoded image is supplied to the baseband processing circuit 43.Then, when the resulting image is supplied to the image processingdevice 21 through the switch 44, the processing device 21 performs theimage processing on the supplied image as the raw-material image 11.

Furthermore, operation of an operation circuit (or “operation unit”)(not illustrated) by the user is provided to the system control circuit54 through the operation system I/F 53, and the system control circuit54 performs control on each block that makes up the digital video camera41 according to the operation by the user.

In the digital video camera 41 that is configured in this manner, theimage processing device 21 can perform the image processing, describedabove, on the image captured by the imaging circuit 42 or the imagealready recorded in the record circuit 50 as the raw-material image 11.

Furthermore, in addition to the digital video camera 41, the presenttechnology can be applied to a mobile terminal equipped with a camera,and the image processing performed by the image processing device 21 maybe provided to the mobile terminal as an application to perform theimage processing in the mobile terminal. Furthermore, the image recordedby the mobile terminal may be transmitted to an application server (acomputer), not illustrated, over a network, and the mobile terminal mayreceive and reproduce the result of the image processing performed inthe application server.

Moreover, each processing that is described referring to the flowchartsdescribed above is not necessarily performed in chronological order ofthe description in the flowchart, and includes the processing that isperformed in parallel or individually (for example, parallel processingor processing by an object). Furthermore, a program may be one that isexecuted by a single CPU and may be one that is executed by the multipleCPUs using distributed processing.

Furthermore, a sequence of processing described above (an informationprocessing method) can be executed in hardware and can be executed insoftware. If the sequence of processing is executed through the use ofsoftware, the program making up that software is installed, from aprogram recording medium on which the program is recorded, into acomputer that is built into dedicated hardware or, for example, into ageneral purpose personal computer that, when various programs areinstalled, can execute various functions.

FIG. 11 is a block diagram illustrating a configuration example of thehardware of the computer that executes the sequence of processingdescribed above using the program.

In the computer, a central processing unit (CPU) 101, a read only memory(ROM) 102, a random access memory 103 are connected to one anotherthrough a bus 104.

An input and output interface 105 is connected to the bus 104. To theinput and output interface 105, are connected an input device 106 thatis made from a keyboard, a mouse, or a microphone, an output circuit 107(or “output unit” 107) that is made from a display or a speaker, astorage circuit 108 (or “storage unit” 108) that is made from a harddisk or a nonvolatile memory, a communication circuit 109 (or“communication unit” 109) that is made from a network interface, and adrive 110 that drives a removable medium 111 such as a magnetic disk, anoptical disk, an optical magnetic disk, or a semiconductor memory.

In the computer that is configured as described above, the CPUT 101loads the program that is stored, for example, in the storage circuit108, onto the RAM 103 through the input and output interface 105 and thebus 104 in order to execute the program. Thus, the sequence ofprocessing described above is performed.

The program executed by the computer (the CPU 101) is recorded in theremovable medium 111 that is a package medium that is made from, forexample, the magnetic disk (including a flexible disk), the optical disk(compact disk-read only memory (CD-ROM), a digital versatile disk (DVD),and the like), the optical magnetic disk, or the semiconductor memory,or is provided via wireless or cable transmission medium such as a localarea network, the Internet, and digital satellite broadcasting.

Then, the program can be installed into the storage circuit 108 throughthe input and output interface 105 by inserting the removable medium 111into the drive 110. Furthermore, the program can be received with thecommunication circuit 109 through the cable or wireless transmissionmedium and be installed on the storage circuit 108. In addition, theprogram can be installed in advance in the ROM 102 or the storagecircuit 108.

Moreover, the present technology can have the following configurations.

(1) An image processing device, including a foreground selectionprocessing circuit to select at least one foreground image that has beenseparated from a source image; a background selection circuit to selectat least two display background images from at least one backgroundimage that has been separated from the source image; and a combinationcircuit to combine the at least one selected foreground image with theat least two display background images to generate a plurality ofcombined images, wherein at least one of the plurality of combinedimages does not appear in the source image.

(2) The device as recited in (1), further including a separation circuitto separate the source image into the at least one foreground image andthe at least one background image.

(3) The device as recited in (1), wherein the source image is a stillimage.

(4) The device as recited in (1), wherein the source image is a movingimage.

(5) The device as recited in (1), wherein the source image is a stillimage formed by a plurality of images.

(6) The device as recited in (1), wherein the plurality of combinedimages make up a moving image.

(7) The device as recited in (1), wherein the at least two displaybackground images are each a portion of a still background image.

(8) The device as recited in (1), wherein the at least two displaybackground images are images included in a series of images that make upa moving image.

(9) The device as recited in (1), wherein the foreground selectioncircuit selects a most recently selected foreground image as a currentlyselected foreground image.

(10) The device as recited in (1), wherein the foreground selectioncircuit selects a best foreground image as a currently selectedforeground image.

(11) The device as recited in (1), wherein the foreground selectioncircuit selects a foreground image based on user input.

(12) The device as recited in (1), wherein the foreground selectioncircuit selects a foreground image automatically.

(13) The device as recited in (12), wherein the foreground selectioncircuit selects at least one foreground image baaed on at least onecriteria selected from the group consisting of whether or not theforeground image is blurred, whether or not a subject of the foregroundimage is smiling, whether or not a subject of the foreground image hasclosed eyes, and the brightness of the foreground image.

(14) The device as recited in (1), further including a determinationprocessing circuit for determining at least one of a size and a positionof a display region for use in selecting a display background image fromthe at least one background image.

(15) The device as recited in (1), wherein the source image is a movingimage and the plurality of combined images make up a moving image.

(16) The device as recited in (1), further including a memory forstoring the plurality of combined images.

(17) The device as recited in (1), wherein the total number of theplurality of combined images is a predetermined number.

(18) The device as recited in (17), further including a determinationprocessing circuit for determining, based on the predetermined number,at least one of a size and a position of a display region for use inselecting a display background image from the at least one backgroundimage.

(19) The device as recited in (1), wherein the device is incorporated ina camera, the camera including an imaging circuit and a display.

(20) The device as recited in (1), wherein the source image is a movingimage, at lease two foreground images are selected, and the plurality ofcombined images make up a moving image in which the at least twoselected foreground images make up a foreground moving image and the atleast two display background images make up a background moving image,and in which, at least one of the foreground moving image and thebackground moving image is reproduced at a speed that is different froma reproduction speed of the source image.

(21) The device as recited in (20), wherein a reproduction frame rate ofthe foreground moving image is different from a reproduction frame rateof the background moving image.

(22) An image processing method, including selecting at least oneforeground image that has been separated from a source image; selectingat least two display background images from at least one backgroundimage that has been separated from the source image; and combining theat least one selected foreground image with the at least two displaybackground images to generate a plurality of combined images, wherein atleast one of the plurality of combined images does not appear in thesource image.

(23) A non-transitory computer-readable medium storing acomputer-readable program for implementing an image processing method,the method including selecting at least one foreground image chat hasbeen separated from a source image; selecting at least two displaybackground images from at least one background image that has beenseparated from the source image; and combining the at least one selectedforeground image with the at least two display background images togenerate a plurality of combined images, wherein at least one of theplurality of combined images does not appear in the source image.

Moreover, the present technology can have the following configurations.

(1) An image processing device including a separation unit that,according to an imaging target object being imaged into a raw-materialimage, separates the raw-material image into a foreground image and abackground image, an extraction unit that sets a display region whichspecifics a region which is defined as a display target with respect tothe background image, and that extracts one part of the foreground imagealong the display region, and a combination unit that combines theforeground image with respect to the background image extracted by theextraction unit.

(2) The image processing device according to (1) in which, when theraw-material image is a still image, the extraction unit extracts onepart of the background image as the display background image whilemoving the display region with respect to the background image, and inwhich the combination unit combines the extracted display backgroundimage with the foreground image.

(3) The image processing device according to (1) or (2), furtherincluding a determination unit that determines the foreground image thatis combined with the display background image, based on the multipleforeground images that are separated from multiple sheets of stillimages that make up the moving image, wherein the raw-material image isa moving image.

(4) The image processing device according to any one of (1) to (3)wherein when the imaging target object that is defined as the foregroundimage is not imaged in the moving image as the raw-material image, thedetermination unit determines the foreground image into which theimaging target object is imaged for the last time, as the foregroundimage that is to be combined with the display background image.

(5) The image processing device according to any one of (1) to (4) inwhich the determination unit selects the foreground image into which theimaging target object is excellently imaged, as the foreground imagethat is to be combined with the display background image.

(5) The image processing device according to any one of (1) to (5),further including a determination unit that determines the foregroundimage and the background image that are combined with respect to themultiple foreground images that are separated from multiple sheets ofstill images that make up the moving image and the multiple displaybackground images, respectively, according to reproduction speeds towhich the foreground image and the background image are set,respectively, when the raw-material image is a moving image.

It should be noted that the present disclosure is not limited to theembodiments described above, and can be variously modified within ascope not departing from the gist of the present disclosure.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2012-248331 filed in theJapan Patent Office on Nov. 12, 2012, the entire contents of which arehereby incorporated by reference.

What is claimed is:
 1. An image processing device, comprising; aforeground selection processing circuit to select at least oneforeground image that has been separated from a source image; abackground selection circuit to select at least two display backgroundimages from at least one background image that has been separated fromthe source image; and a combination circuit to combine the at least oneselected foreground image with the at least two display backgroundimages to generate a plurality of combined images, wherein at least oneof the plurality of combined images does not appear in the source image.2. The device as recited in claim 1, further comprising a separationcircuit to separate the source image into the at least one foregroundimage and the at least one background image.
 3. The device as recited inclaim 1, wherein the source image is a still image.
 4. The device asrecited in claim 1, wherein the source image is a moving image.
 5. Thedevice as recited in claim 1, wherein the source image is a still imageformed by a plurality of images.
 6. The device as recited in claim 1,wherein the plurality of combined images make up a moving image.
 7. Thedevice as recited in claim 1, wherein the at least two displaybackground images are each a portion of a still background image.
 8. Thedevice as recited in claim 1, wherein the at least two displaybackground images are images included in a series of images that make upa moving image.
 9. The device as recited in claim 1, wherein theforeground selection circuit selects a most recently selected foregroundimage as a currently selected foreground image.
 10. The device asrecited in claim 1, wherein the foreground selection circuit selects abest foreground image as a currently selected foreground image.
 11. Thedevice as recited in claim 1, wherein the in foreground selectioncircuit selects a foreground image based on user input.
 12. The deviceas recited in claim 1, wherein the foreground selection circuit selectsa foreground image automatically.
 13. The device as recited in claim 12,wherein the foreground selection circuit selects at least one foregroundimage based on at least one criteria selected from the group consistingof whether or not the foreground image is blurred, whether or not asubject of the foreground image is smiling, whether or not a subject ofthe foreground image has closed eyes, and the brightness of theforeground image.
 14. The device as recited in claim 1, furthercomprising a determination processing circuit for determining at leastone of a size and a position of a display region for use in selecting adisplay background image from the at least one background image.
 15. Thedevice as recited in claim 1, wherein the source image is a moving imageand the plurality of combined images make up a moving image.
 16. Thedevice as recited in claim 1, further comprising a memory for storingthe plurality of combined images.
 17. The device as recited in claim 1,wherein the total number of the plurality of combined images is apredetermined number.
 18. The device as recited in claim 17, furthercomprising a determination processing circuit for determining, based onthe predetermined number, at least one of a size and a position of adisplay region for use in selecting a display background image from theat least one background image.
 19. The device as recited in claim 1,wherein the device is incorporated in a camera, the camera comprising animaging circuit and a display.
 20. The device as recited in claim 1,wherein the source image is a moving image, at least two foregroundimages are selected, and the plurality of combined images make up amoving image in which the at least two selected foreground images makeup a foreground moving image and the at least two display backgroundimages make up a background moving image, and in which, at least one orthe foreground moving image and the background moving image isreproduced at a speed that is different from a reproduction speed of thesource image.
 21. The device as recited in claim 20, wherein areproduction frame rate of the foreground moving image is different froma reproduction frame rate of the background moving image.
 22. An imageprocessing method, comprising: selecting at least one foreground imagethat has separated from a source image; selecting at least two displaybackground images from at least one background image that has beenseparated from the source image; and combining the at least one selectedforeground image with the at least two display background images togenerate a plurality of combined images, wherein at least one of theplurality of combined images does not appear in the source image.
 23. Anon-transitory computer-readable medium storing a computer-readableprogram for implementing an image processing method, the methodcomprising: selecting at least one foreground image that has beenseparated from a source image; selecting at least two display backgroundimages from at least one background image that has been separated fromthe source image; and combining the at least one selected foregroundimage with the at least two display background images to generate aplurality of combined images, wherein at least one of the plurality ofcombined images does not appear in the source image.